A. Field of the Invention
The field of the present invention is video compression.
B. Background
The present invention relates to the encoding, formatting, and distribution of compressed video programs. It is particularly, but not necessarily, applicable to a particular video compression technique known as variable bit-rate(VBR) encoding. VBR encoding can be used to overcome the well-known problem of most video compression encoders where the perceived image quality tends to vary as a function of image complexity. Typically, a video program will contain a variety of scenes. Many of these scenes are lacking in motion or detail and are therefore easily compressed, while many other scenes contain complex details which are generally more difficult to compress, particularly when moving in a complex or random manner. Therefore, unless the available bandwidth is very high, the perceived quality of the decompressed and reconstructed images will tend to vary from one scene to the next. This problem becomes more serious as the available bandwidth is reduced until, eventually, the video quality becomes unacceptable, often because of just a few problem scenes.
VBR encoding overcomes this problem by allocating more bits to those scenes which are difficult to compress and fewer bits to those scenes which are more easily compressed. In this way, the decompressed and reconstructed images can be made to appear consistently uniform, and therefore superior to the reconstructed images derived from the constant bit-rate encoder adjusted for the same average rate of compression. As a result, it is possible to compress a video program more efficiently when using the VBR encoding technique. This not only increases the number and variety of programs or program streams that can be delivered over a fixed-bandwidth communication channel, but also reduces the storage capacity requirements at the head end or other site where the program library is maintained.
The disadvantage of the VBR encoding technique is that it presents certain problems when manipulating or editing the compressed bit-streams. In particular, it becomes difficult to efficiently utilize a fixed-bandwidth communication channel since the variable bit-rate stream may at times exceed the capacity of the channel, while at other times, it may utilize only a fraction of the available channel capacity. As would be clear to one in the art, the possibility of exceeding the capacity of the channel is not only a problem with VBR encoded data. The present invention is applicable also to non-VBR encoded data.
One known technique that is used to alleviate this problem is to buffer the compressed bit-stream at the transmission end of the communication channel in order to convert the variable rate stream to a constant rate stream. In such a case, it is also necessary to buffer the signal received at the other end of the channel in order to recover the variable rate stream that is necessary for proper timing of the reconstructed video images. Unfortunately, the required amount of buffering would be prohibitively expensive and would introduce long delays into the distribution system. Moreover, existing video compression standards such as the MPEG standards, a set of International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) standards, specify limits on the amount of buffering required for conforming decoders. Therefore, it is important that the received bitstreams be decodable without exceeding these limits. That is, it is important to find methods for reducing the decoder memory requirements.
Another prior art technique that can be used to reduce the inefficiency of transmitting VBR encoded (and non-VBR encoded) programs over a fixed-bandwidth channel combines a plurality of program streams into a single multiplex. Although each additional program stream will increase the overall data rate of the multiplex, the variance of the average per stream data rate of this multiplex will tend to decrease in approximate proportion to the number of program streams, assuming approximate statistical independence of them. Therefore, if the nominal rate that can be accommodated by the channel is significantly greater than the average rate of a single program stream, then the channel utilization can be significantly improved. This technique is known in the art as statistical multiplexing.
When using buffering, one prior art method of assuring that buffer overflow does not occur is to add feedback between the encoder buffer and the encoder. When the buffer approaches a full state, a buffer fullness signal from the buffer informs the encoder to reduce the bit-rate so that the buffer does not overflow. When the buffer has more room; the feedback signal, from the buffer to the encoder, enables the encoder to increase the bit-rate in order to maintain image quality. Such feedback is particularly effective when combined with statistical multiplexing. A single encoder buffer may be used, or several buffers. In the case of a single buffer, it would be at the output of the multiplexer, and feedback from that buffer would be to all the encoders of the programs being multiplexed. Because the multiplexing reduces effective per stream data rate variance, in general, less memory would be required to convert the multiplex to a fixed bit-rate stream than if each individual encoder included a VBR-to-constant bit-rate buffer.
Statistical multiplexing with feedback for conveying encoded, particularly VBR encoded program streams over fixed-bandwidth channels, can be effective, but generally may not be sufficient when uniform perceived image quality is important.
In addition, there are often situations where no feedback is possible between the output buffer of the multiplexer and the individual encoders. One such situation occurs when multiplexing already encoded data streams. Another occurs when the encoders are located in an area physically remote from the multiplexer. Both these situations are referred to herein as remote encoding, indicating that encoding and multiplexing are remote in time, location, or otherwise, so that no feedback is possible from the multiplexer to the encoders of the program streams to be multiplexed.
In addition, one of the factors that limits the efficiency of transmitting the multiplexed bit stream is the finite overall memory available for buffering at the decoder/demultiplexer. While statistical multiplexing usually provides sufficient benefit due to the small probability that all program streams will have complex scenes at the same time, there is still a finite chance of such data rate peaks occurring. To accommodate these peaks, long delays, and therefore large buffers, must be designed into the decoders in order to prevent the decoder buffers from underflowing when the capacity of the channel is insufficient to deliver the data in time for decoding. The opposite extreme occurs when all of the programs contain scenes which are easily compressed. In this case, data can be delivered too quickly and the decoder buffers may overflow unless fill packets, containing stuffing bytes, are inserted into the bit stream. This reduces channel utilization (channel efficiency) since the fill packets are not useful for decoding and are discarded at the receivers.
Our co-pending U.S. patent application entitled "Method and Apparatus for Multiplexing Video Programs For Improved Channel Utilization," filed concurrently with the present application, incorporated herein by reference, and hereinafter referred to as "Our Multiplexing Invention," discloses a multiplexing method for determining the order of forming the multiplex that assures that no overflow occurs in any decoder buffer, thereby increasing the efficiency of transmitting the multiplex. Even when using the method of Our Multiplexing Invention, there is still a finite chance that all program streams have scenes at the same time of such complexity and that peaks in the decoder buffer memory requirement are such that no more decoder buffer space is available. A method for reducing such data rate fluctuations is disclosed in our co-pending U.S. patent application entitled "Method and Apparatus for Increasing Channel Utilization for Digital Video Transmission," filed concurrently with the present application, incorporated herein by reference, and hereinafter referred to as "Our Channel Utilization Invention." The method of Our Channel Utilization Invention comprises varying the formation of the multiplex of the different program streams to reduce the memory requirement of the decoder buffers, varying including adjusting the relative alignment of the program streams. Different embodiments of adjusting include inserting program material such as a pause at selected points in time, causing decoder buffer underflow at selected points in time, and varying the rate of decoding and display at selected points in time.
The methods of Our Multiplexing Invention and Our Channel Utilization Invention do not reduce the quality of the pictures transmitted. There may still be situations in which these methods are not sufficient or cannot be used. In such situations, it may be necessary to reduce the amount of data at selected points in time in order to avoid exceeding the capacity of the channel.
Thus, there is a need in the art for technology that can be used to modify encoded video programs, in particular VBR-encoded programs, in order to reduce the data rate fluctuations in such programs without significantly affecting the image quality.
Thus, there also is a need for technology for modifying encoded video that reduces data rate fluctuations when transmitting a stream of a multiplex of several encoded video programs, applicable to both real time encoding and remote encoding situations.
Thus, there also is a need for technology for modifying encoded video that eliminates the chance of a buffer overflow condition occurring at the decoder.
Thus, there also is a need for technology for modifying encoded video that maximizes channel efficiency.
Thus, there also is a need for technology for delivering the information that is removed from the encoded video by modifying to certain receivers capable of extracting and utilizing the information to deliver improved image quality.